{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "<a href=\"https://colab.research.google.com/drive/1Ib3QcE9TgUZHuYXgepwk9FS-eSbDniQ8?usp=sharing\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
        "\n",
        "### 🛠️ Automatic Prompt Engineer (APE)\n",
        "\n",
        "APE is a method for automatically generating and optimizing prompts for language models.\n",
        "\n",
        "### Key points:\n",
        "\n",
        "* 🔑 **Core concept:** Use AI to create and refine prompts, reducing manual engineering effort.\n",
        "\n",
        "* ⚙️ **Process:**\n",
        "  * Generate candidate prompts\n",
        "  * Evaluate prompt performance\n",
        "  * Iteratively optimize prompts\n",
        "\n",
        "* 🛠️ **Components:**\n",
        "  * Prompt generator\n",
        "  * Performance evaluator\n",
        "  * Optimization algorithm\n",
        "\n",
        "* 🌟 **Advantages:**\n",
        "  * Discovers effective prompts automatically\n",
        "  * Adapts to different tasks and model architectures\n",
        "\n",
        "* 💼 **Applications:**\n",
        "  * Task-specific prompt optimization\n",
        "  * Improving model performance across various domains\n",
        "\n",
        "* 🚀 **Implementation:**\n",
        "  * Define task and evaluation metrics\n",
        "  * Use large language models to generate initial prompts\n",
        "  * Apply optimization techniques (e.g., genetic algorithms, gradient-based methods)\n",
        "\n",
        "* ⚖️ **Challenges:**\n",
        "  * Balancing exploration and exploitation in prompt space\n",
        "  *Ensuring generated prompts are interpretable and safe\n",
        "  \n",
        "* 🔄 **Variations:**\n",
        "  * Multi-task APE: Optimize prompts for multiple related tasks\n",
        "  * Constrained APE: Generate prompts within specific guidelines or structures"
      ],
      "metadata": {
        "id": "Y2eqHW0TK87s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q -U langchain-groq==0.2.4"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XNrPX8MZJm9e",
        "outputId": "9e335ca2-7773-4825-b631-17bf7ae15527"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/121.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.9/121.9 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_groq import ChatGroq\n",
        "from langchain.prompts import ChatPromptTemplate\n",
        "from IPython.display import display, Markdown"
      ],
      "metadata": {
        "id": "gOLahJCEJv6_"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import getpass"
      ],
      "metadata": {
        "id": "I6qvHWQsJxWq"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 🔑 Provide Groq API Key\n",
        "\n",
        "- [Groq API Key](https://console.groq.com/keys)\n"
      ],
      "metadata": {
        "id": "GaoPiWagJyoV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "os.environ[\"GROQ_API_KEY\"] = getpass.getpass()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f-Ioab8HJ037",
        "outputId": "5222038f-3c74-41e1-b844-dd774ab03353"
      },
      "execution_count": 4,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "llm = ChatGroq(\n",
        "    model=\"llama3-8b-8192\",\n",
        "    temperature=0.5\n",
        ")"
      ],
      "metadata": {
        "id": "RzCcry2GJ2Z1"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import re\n",
        "\n",
        "# APE Prompt Generation\n",
        "ape_generation_prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", \"You are an AI specialized in creating effective prompts for language models.\"),\n",
        "        (\"human\", \"\"\"Task: Create a prompt that will help a language model perform the following task effectively:\n",
        "\n",
        "          {task}\n",
        "\n",
        "          Generate 3 different prompts for this task. Each prompt should be designed to elicit a high-quality\n",
        "\n",
        "          response from a language model. Consider different angles, formats, and instructions that might lead to better results.\n",
        "\n",
        "          Your response should be in the following format:\n",
        "\n",
        "          Prompt 1:\n",
        "          [Your first prompt here]\n",
        "\n",
        "          Prompt 2:\n",
        "          [Your second prompt here]\n",
        "\n",
        "          Prompt 3:\n",
        "          [Your third prompt here]\n",
        "\n",
        "          Generated Prompts:\"\"\")\n",
        "    ]\n",
        ")\n",
        "\n",
        "# APE Evaluation\n",
        "ape_evaluation_prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", \"You are an AI specialized in evaluating the effectiveness of prompts for language models.\"),\n",
        "        (\"human\", \"\"\"Evaluate the following prompts for their effectiveness in accomplishing this task:\n",
        "\n",
        "          Task: {task}\n",
        "\n",
        "          {prompts}\n",
        "\n",
        "          For each prompt, provide a score from 1-10 and a brief explanation of its strengths and weaknesses.\n",
        "\n",
        "          Consider factors such as clarity, specificity, and potential to elicit high-quality responses.\n",
        "\n",
        "          Your evaluation should be in the following format:\n",
        "\n",
        "          Prompt 1:\n",
        "          Score: [score]/10\n",
        "          Evaluation: [Your evaluation here]\n",
        "\n",
        "          Prompt 2:\n",
        "          Score: [score]/10\n",
        "          Evaluation: [Your evaluation here]\n",
        "\n",
        "          Prompt 3:\n",
        "          Score: [score]/10\n",
        "          Evaluation: [Your evaluation here]\n",
        "\n",
        "          Your evaluation:\"\"\")\n",
        "    ]\n",
        ")\n",
        "\n",
        "def generate_prompts(task):\n",
        "    chain = ape_generation_prompt | llm\n",
        "    return chain.invoke({\"task\": task}).content\n",
        "\n",
        "def evaluate_prompts(task, prompts):\n",
        "    chain = ape_evaluation_prompt | llm\n",
        "    return chain.invoke({\"task\": task, \"prompts\": prompts }).content\n",
        "\n",
        "def test_prompt(prompt, task):\n",
        "    test_prompt = ChatPromptTemplate.from_messages(\n",
        "      [\n",
        "          (\"system\", \"You are an AI assistant completing a given task.\"),\n",
        "          (\"human\", \"{prompt}\\n\\nTask: {task}\")\n",
        "      ]\n",
        "    )\n",
        "    chain = test_prompt | llm\n",
        "    return chain.invoke({\"task\": task, \"prompt\": prompt }).content\n",
        "\n",
        "def parse_prompts(generated_prompts):\n",
        "    prompts = re.findall(\n",
        "        r\"Prompt \\d+:\\n(.*?)(?=\\n\\nPrompt \\d+:|$)\", generated_prompts, re.DOTALL\n",
        "    )\n",
        "    return [prompt.strip() for prompt in prompts if prompt.strip()]\n",
        "\n",
        "\n",
        "def parse_scores(evaluation):\n",
        "    scores = re.findall(r\"Score: (\\d+)/10\", evaluation)\n",
        "    return [int(score) for score in scores if score.isdigit()]\n",
        "\n",
        "\n",
        "def ape_process(task, iterations=2):\n",
        "    best_prompt = \"\"\n",
        "    best_score = 0\n",
        "\n",
        "    for i in range(iterations):\n",
        "        print(f\"Iteration {i+1}\")\n",
        "\n",
        "        # Generate prompts\n",
        "        generated_prompts = generate_prompts(task)\n",
        "        display(Markdown(f\"**Generated Prompts:**\\n{generated_prompts}\"))\n",
        "\n",
        "        # Evaluate prompts\n",
        "        evaluation = evaluate_prompts(task, generated_prompts)\n",
        "        display(Markdown(f\"**Prompt Evaluation:**\\n{evaluation}\"))\n",
        "\n",
        "        # Parse prompts and scores\n",
        "        prompts = parse_prompts(generated_prompts)\n",
        "        scores = parse_scores(evaluation)\n",
        "\n",
        "        # Ensure we have valid prompts and scores\n",
        "        if prompts and scores:\n",
        "            # Make sure we have the same number of prompts and scores\n",
        "            min_length = min(len(prompts), len(scores))\n",
        "            prompts = prompts[:min_length]\n",
        "            scores = scores[:min_length]\n",
        "\n",
        "            if max(scores) > best_score:\n",
        "                best_score = max(scores)\n",
        "                best_prompt = prompts[scores.index(max(scores))]\n",
        "\n",
        "            print(f\"Best prompt so far (score {best_score}/10):\")\n",
        "            print(best_prompt)\n",
        "        else:\n",
        "            print(\"Failed to generate valid prompts or scores in this iteration.\")\n",
        "\n",
        "        print()\n",
        "\n",
        "    # If we didn't find a good prompt, use a default one\n",
        "    if not best_prompt:\n",
        "        best_prompt = f\"Please {task}\"\n",
        "        print(\"Using default prompt due to generation issues.\")\n",
        "\n",
        "    # Test the best prompt\n",
        "    final_result = test_prompt(best_prompt, task)\n",
        "    return best_prompt, final_result"
      ],
      "metadata": {
        "id": "RCjbg-xkJ4cY"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Example usage\n",
        "task = \"Explain the concept of quantum entanglement to a 10-year-old.\"\n",
        "\n",
        "best_prompt, final_result = ape_process(task)\n",
        "\n",
        "display(\n",
        "    Markdown(\n",
        "        f\"\"\"**Final Best Prompt:**\n",
        "{best_prompt}\n",
        "\n",
        "**Final Result:**\n",
        "{final_result}\"\"\"\n",
        "    )\n",
        ")\n"
      ],
      "metadata": {
        "id": "YCnIpHUSKWsT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "c1c8b8e2-6ea7-4e38-e593-70bf772ba856"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Iteration 1\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Generated Prompts:**\nHere are three different prompts to help a language model explain quantum entanglement to a 10-year-old:\n\nPrompt 1:\nExplain quantum entanglement in a way that a 10-year-old could understand. Use simple language and avoid technical jargon. Imagine you're telling a friend about a cool science concept, and make it sound exciting and fun!\n\nPrompt 2:\nWrite a story about two toy cars that are \"entangled\" in a way that's similar to quantum entanglement. How do they behave when they're connected, and what happens when they're separated? Use this analogy to help explain the concept of entanglement in a way that's relatable and easy to grasp.\n\nPrompt 3:\nImagine you have a special kind of walkie-talkie that lets you talk to your friend who's on the other side of the world, but it works in a weird way. When you say something into the walkie-talkie, your friend hears it instantly, even if you're thousands of miles apart. How does it work? Use this thought experiment to help explain the concept of quantum entanglement, and make sure to highlight the strange and amazing implications of this phenomenon."
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Prompt Evaluation:**\nHere are my evaluations for each prompt:\n\nPrompt 1:\nScore: 7/10\nEvaluation: This prompt is clear and concise, and it provides a good starting point for the language model to explain quantum entanglement in simple language. The instruction to use \"simple language and avoid technical jargon\" is helpful, and the phrase \"make it sound exciting and fun\" encourages the model to be engaging and enthusiastic. However, the prompt is quite general and doesn't provide a specific framework or analogy for the model to work with, which might lead to a response that is more of a general explanation rather than a concrete example.\n\nPrompt 2:\nScore: 9/10\nEvaluation: This prompt is excellent! It provides a specific and relatable analogy that can help explain the concept of entanglement in a way that's easy to grasp. The idea of two toy cars being \"entangled\" is a great way to illustrate the phenomenon, and the instruction to use this analogy to explain entanglement provides a clear framework for the model to work with. The prompt also encourages the model to think creatively and come up with a story that's engaging and fun. The only potential weakness is that the analogy might not fully capture the complexities of quantum entanglement, but overall, this prompt is well-designed.\n\nPrompt 3:\nScore: 8.5/10\nEvaluation: This prompt is very effective! It provides a thought-provoking analogy that can help explain the concept of entanglement in a way that's easy to understand. The idea of a special walkie-talkie that lets you talk to your friend instantly, no matter the distance, is a great way to illustrate the phenomenon of entanglement. The prompt also encourages the model to think creatively and come up with a explanation that's engaging and fun. The only potential weakness is that the analogy might be a bit too complex for a 10-year-old, and the model might need to simplify the explanation further. However, overall, this prompt is well-designed and has the potential to elicit a high-quality response."
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Best prompt so far (score 9/10):\n",
            "Write a story about two toy cars that are \"entangled\" in a way that's similar to quantum entanglement. How do they behave when they're connected, and what happens when they're separated? Use this analogy to help explain the concept of entanglement in a way that's relatable and easy to grasp.\n",
            "\n",
            "Iteration 2\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Generated Prompts:**\nHere are three prompts designed to help a language model explain quantum entanglement to a 10-year-old:\n\nPrompt 1:\nExplain quantum entanglement in a way that a 10-year-old can understand, using simple language and relatable examples. Imagine you're explaining it to a friend who has never heard of it before. Make sure to include a fun analogy or metaphor to help them grasp the concept.\n\nPrompt 2:\nWrite a story about two toy cars, \"Racing Ruby\" and \"Speedy Sam,\" that are connected by a special kind of invisible string. When something happens to one car, it instantly affects the other car, even if they're on opposite sides of the room. Use this story to explain the concept of quantum entanglement, focusing on the idea that particles can be connected in a way that lets them affect each other instantly, no matter how far apart they are.\n\nPrompt 3:\nImagine you're a scientist who has just discovered a magical box that can connect two objects in a special way. When you put a toy in the box with a partner toy, they become \"entangled\" and can affect each other in strange and amazing ways. Write a report to a 10-year-old describing the magical box and its powers, using simple language and examples to explain how it works. Be sure to include some cool and surprising facts about the box and its effects on the entangled objects."
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Prompt Evaluation:**\nHere are my evaluations for each prompt:\n\nPrompt 1:\nScore: 8/10\nEvaluation: This prompt is clear and concise, providing a good foundation for the language model to explain quantum entanglement in a simple and relatable way. The instruction to use a fun analogy or metaphor is particularly helpful, as it encourages the model to be creative and engaging. However, the prompt could be improved by providing more specific guidance on what kind of examples or analogies would be most effective for a 10-year-old.\n\nPrompt 2:\nScore: 9/10\nEvaluation: This prompt is highly effective, as it uses a storytelling approach to explain a complex concept in a way that is both engaging and easy to understand. The use of relatable characters and a concrete scenario helps to make the concept of entanglement more accessible to a 10-year-old. The prompt also provides a clear structure for the language model to follow, making it more likely to produce a high-quality response. The only potential weakness is that the prompt may be a bit too focused on the toy cars, which could limit the model's ability to provide additional context or explanations.\n\nPrompt 3:\nScore: 7/10\nEvaluation: This prompt is a good attempt to explain quantum entanglement using a fictional scenario, but it could be improved in several ways. The instruction to write a report to a 10-year-old is a bit vague, and the prompt could benefit from more specific guidance on what kind of information to include and how to present it. Additionally, the magical box analogy, while creative, may not be the most effective way to explain entanglement, as it may be perceived as too fantastical or misleading. However, the prompt does provide a clear structure and encourages the model to use simple language and examples, which are important for explaining complex concepts to a 10-year-old."
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Best prompt so far (score 9/10):\n",
            "Write a story about two toy cars that are \"entangled\" in a way that's similar to quantum entanglement. How do they behave when they're connected, and what happens when they're separated? Use this analogy to help explain the concept of entanglement in a way that's relatable and easy to grasp.\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**Final Best Prompt:**\nWrite a story about two toy cars that are \"entangled\" in a way that's similar to quantum entanglement. How do they behave when they're connected, and what happens when they're separated? Use this analogy to help explain the concept of entanglement in a way that's relatable and easy to grasp.\n\n**Final Result:**\nOnce upon a time, in a toy box filled with all sorts of fun and colorful toys, there lived two best friend toy cars, Zoom and Zoomster. They were identical, shiny, and loved to speed around the room, racing each other and having the best of times.\n\nOne day, a curious scientist, Dr. Toybox, discovered that Zoom and Zoomster had a special connection. Whenever they were placed together, they would start to behave in a very strange and amazing way.\n\nWhen Zoom moved to the left, Zoomster would instantly move to the right, even if they were on opposite sides of the room! It was as if they were connected by an invisible string. But here's the really cool part: when Zoomster moved to the right, Zoom would instantly move to the left, even if they were still on opposite sides of the room!\n\nDr. Toybox was fascinated by this phenomenon and decided to test it further. She placed Zoom and Zoomster in separate boxes, so they couldn't see or touch each other. Then, she had someone move Zoom to the left while someone else moved Zoomster to the right.\n\nGuess what? Even though they were separated by miles, Zoom and Zoomster still reacted to each other's movements! When Zoom moved left, Zoomster moved right, and vice versa. It was as if they were still connected by that invisible string.\n\nThis is similar to what happens with quantum entanglement. When two tiny particles, like atoms or electrons, are connected in a special way, they can affect each other even if they're really far apart. It's like they're \"talking\" to each other in a secret language that's faster than the speed of light.\n\nImagine you have two toy cars, just like Zoom and Zoomster. If they're entangled, whenever one car moves, the other car will move instantly, no matter where it is in the world. It's like they're connected by a magic string that lets them communicate with each other.\n\nBut here's the really cool part: even if you separate the cars and take them to opposite sides of the universe, they'll still be connected. If you make one car spin around, the other car will spin around too, even if it's on the other side of the universe!\n\nSo, to sum it up, quantum entanglement is like having two toy cars that are connected in a special way. When they're together, they can affect each other instantly, and even when they're really far apart, they can still \"talk\" to each other in a secret language. It's a really cool and weird phenomenon that scientists are still trying to understand better!\n\nI hope that helps you understand quantum entanglement, kiddo!"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "kY1AoektWzh_"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}